1.2 Learning as Agent with Large-Scale RL

OpenAI (2025):

Deep research independently discovers, reasons about, and consolidates insights from across the web.

To accomplish this, it was trained on real-world tasks requiring browser and Python tool use,

using the same reinforcement learning methods behind OpenAI o1, our first reasoning model.

Check OpenAI Deep Research’s demo video for more details.

1.3 Future: Agent + Tool Protocol = Versatile

2.2 More Complex: RL with LLMs

RL with Large Language Models (LLMs) is even more challenging.

2.3 Even More Complex: Flexible & Efficient Implementation

3.2 Trade-off: Single-Controller or Multi-Controller?

Which paradigm should we choose?

3.3 New Paradigm: Hybrid-Controller!

Hybrid-Controller = Single-Controller + N x Multi-Controller

3.5.1 Background: Different Workloads Have Different Optimal Parallelism Strategies

The optimal parallelism strategies for different workloads are usually different, e.g.,

• training needs to save gradients & optimizer states other than parameters, requiring more sharding,
• generation has fewer states, allowing less sharding.

3.5.2 Inter-Stage: Hybrid Engine

3.5.3 Intra-Stage: Diverse Parallelisms

• ❌ Naive: wrap the batch generation in a for-loop, synchronizing for each turn, thus causing many bubbles.
• ✅ Efficient: utilize the async engine, managing generation at the request level instead of the batch level.

4.2 Efficient RL with Huge MoE like DeepSeek-V3-671B (ETA: Late May’25)

verl is working on supporting efficient RL training for huge MoE like DeepSeek-V3-671B, based on the following features:

1. MoE models with GPTModel class for actor and critic
2. Multi-node inference
3. Parameter sharding manager for Megatron-Core V0.12 + latest version of inference engines

4.3 Agentic RL with Diverse Environments & Tools (Planned)

1. Our ongoing RFC
2. Integrating protocols like MDP
3. Integrating existing environments & tools, e.g. 

• KORGym @ ByteDanceSeed (Shi et al. 2025)
• Atropos @ Nous Research (Dakota Mahan 2025)

4.4 Other Plans

1. Partial Rollout (Kimi Team 2025)
2. Multi-Token-Prediction (MTP) (Gloeckle et al. 2024)
3. …

5.2 DP Balancing

5.2.1 Load Imbalance in DP

• Parallelism usually needs synchronization between different ranks.
• Data Parallelism (DP) like ZeRO is the most commonly used parallelism strategy.
• However, DP performance might be damaged by load imbalance, which is especially severe in long-context training.

5.2.2 Balancing across DP Ranks

• balance the valid tokens dispatched to each rank
• by reordering the samples in each batch

To enable this, use balance_batch.

5.2.3 Balancing across Micro Batches

However, in gradient accumulation,

• it’s not enough to only balance valid tokens in a batch,
• since DP syncs in the unit of micro batch.

To resolve this, verl supports to

• balance the valid tokens across micro batches
• by evenly deviding the data sequences in the batch before packing into micro batches

To enable this, use use_dynamic_bsz.

5.3 Other Features

1. Multi-Model LLMs’ RL
2. Full support for RL with AMD (ROCm Kernel) hardwares
3. Gradient Checkpointing (enable_gradient_checkpointing)
4. Torch Compile (use_torch_compile)
5. Liger Kernel (use_liger)
6. …

You could also customize the field names via config. Please check the data section in config files like ppo_trainer.yaml for more details.

data:
  custom_cls:
    path: null # path to the `.py` file containing the `class` definition
    name: null # the `class` name

--data.custom_cls.path=./examples/dataset/custom_dataset.py \
--data.custom_cls.name=CustomDataset

The custom dataset class defined in the .py file is required to accept the following initialization parameters:

class CustomDataset: # You could also inherit from `RLHFDataset`
  def __init__(
      self,
      data_files: Union[str, List[str]],
      tokenizer: PreTrainedTokenizer,
      config: DictConfig,
      processor: Optional[ProcessorMixin] = None,
  ):
      ...

6.2 Customizing the Reward

verl allows to define custom reward function via the custom_reward_function config:

custom_reward_function:
  path: null # path to the `.py` file containing the function definition
  name: compute_score # the function name after `def`
reward_model:
  reward_manager: naive

--custom_reward_function.path=./examples/reward_fn/custom_reward_fn.py \
--custom_reward_function.name=compute_score \
--reward_model.reward_manager=naive

The function defined in .py should accept the parameters passed from the reward manager __call__ method. Taking NaiveRewardManager as an example:

class NaiveRewardManager:
    def __call__(self, data: DataProto, return_dict: bool=False):
        # Preprocessing for the input data
        score = self.compute_score(
            data_source=data_source,
            solution_str=solution_str,
            ground_truth=ground_truth,
            extra_info=extra_info,
        )
        # Other processing for the final `reward`

6.3 Customizing the Loss Function

To modify the loss function, the most convenient way is to

1. search for the .backward() call
2. modify functions like compute_policy_loss
3. or add loss terms like entropy_loss

For example, the DataParallelPPOActor.update_policy method defines the loss function as follows:

class DataParallelPPOActor(BasePPOActor):
    def update_policy(self, data: DataProto):
        pg_loss = compute_policy_loss(
            old_log_prob=old_log_prob, log_prob=log_prob,
            advantages=advantages, # ...
        )
        entropy_loss = agg_loss(loss_mat=entropy)
        policy_loss = pg_loss - entropy_loss * entropy_coeff
        kld = kl_penalty(
            logprob=log_prob, ref_logprob=ref_log_prob, # ...
        )
        kl_loss = agg_loss(loss_mat=kld)
        policy_loss = policy_loss + kl_loss * self.config.kl_loss_coef
        loss.backward()

6.4 Customizing the Training Logic

As mentioned above, the main training logic is concentrated in the fit function of the trainer classes like RayPPOTrainer.

For example, the DAPORayTrainer class overrides the fit function to implement the “dynamic sampling” feature:

(See the next slide for the code ➡️)